[core] Fix schema evolution for multiset elements - #9545
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
[P1] Support reading existing multiset data after committing the schema change
This change lets SchemaManager commit a multiset element-type evolution, but the read-side schema-evolution path still has no MultisetType support. SchemaEvolutionUtil.createCastExecutor handles RowType, ArrayType, and MapType; a multiset change falls through to CastExecutors.resolve, which returns null for multiset-to-multiset casts.
I reproduced this on the current PR head with the following sequence:
- Create a table containing
MULTISET<INT>. - Write a row under that schema.
- Alter the element type to
BIGINT. - Read the existing row.
The ALTER succeeds, but the read fails with:
NullPointerException: Cannot cast from type MULTISET<INT> to type MULTISET<BIGINT>
at SchemaEvolutionUtil.createCastExecutor(SchemaEvolutionUtil.java:262)
The same missing read path also affects nested row evolution inside a multiset, such as adding a field or widening an existing field, whenever files written with the old schema are present.
Please add a MultisetType branch to the read-side evolution logic. Since a multiset is represented as an InternalMap, it should recursively cast the key array (the elements) while preserving the integer multiplicity value array. An end-to-end regression test should write data before the schema change and read it afterward; the current tests only verify the resulting metadata.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
The earlier missing read-side multiset cast is fixed: the new write-before-ALTER/read-afterward regression passes for INT to BIGINT and nested row widening/addition. There is still a multiplicity-loss case when an accepted element evolution maps distinct old elements to equal new elements; details are inline.
Validation: 60 SchemaManager/table tests passed after compiling this PR's core sources and matching RESTApi, avoiding the newer cached SchemaManager-interface ABI. An additional real stored-table probe checked three cases: INT to BIGINT preserved total multiplicity 5; INT to FLOAT with colliding values, and dropping a distinguishing field from a multiset's row element, both produced duplicate keys and total multiplicity 3 after the standard row-value copy. This was a local core read/copy reproduction, not a Flink or Spark SQL run.
| createCastExecutor( | ||
| inputType.getElementType(), targetType.getElementType())); | ||
|
|
||
| CastedMap castedMap = CastedMap.fromKey(castElementGetter); |
There was a problem hiding this comment.
[P1] Preserve multiplicities when evolved elements become equal
The new key-only lazy cast keeps one entry per old element even when an allowed schema change makes their keys equal. I reproduced this with real table writes followed by ALTER: MULTISET<INT> containing {16777216:2, 16777217:3} can evolve to FLOAT, yielding two identical float keys; InternalRowUtils.copy then overwrites the first count and returns multiplicity 3 instead of 5. Dropping a from MULTISET<ROW<a INT,b INT>> containing {(10,1):2,(20,1):3} has the same failure. INT to BIGINT is a passing control. These changes are accepted by the current schema manager, so returning duplicate keys violates the multiset representation and loses occurrences through normal materialization. Either combine multiplicities for equal evolved elements, or reject element evolutions that cannot preserve uniqueness before committing the schema. Please add write/ALTER/read-and-copy regressions for these collisions.
Purpose
Tests